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Preliminary  Report  on  Conducting 
SEI-Assisted  Assessments  of  Software 
Engineering  Capability 


Abstract 

Characterizing  the  state  of  software  engineering  practice  within  an  organization  is  a 
necessary  prerequisite  to  orderly,  meaningful,  and  sustainable  improvement  of  the 
organization’s  ability  to  produce  or  support  cost-effective,  high  quality  software  prod¬ 
ucts.  The  Software  Engineering  Institute  (SEI)  is  developing  a  methodology  for  con¬ 
ducting  SEI-assisted  assessments  of  software  engineering  capability.  The  assess¬ 
ment  methodology  has  five  phases:  1)  selecting  the  candidate  organization,  2)  pre¬ 
paring  for  the  assessment,  3)  conducting  the  assessment,  4)  communicating  final 
assessment  findings  and  action  recommendations,  and  5)  post-assessment  follow¬ 
up  activities.  This  report  describes  the  methodology  in  detail. 


1.  Introduction 

Process  assessments  are  used  to  study  software  organizations  and  determine  the  state  of  their 
development  and  maintenance  processes.  The  intent  of  assessments  is  to  understand  the  state 
of  practice  of  software  engineering  in  the  organization,  to  identify  key  areas  for  improvement,  and 
to  initiate  actions  that  will  lead  to  improvements.  The  SEI  is  developing  an  assessment  instru¬ 
ment  that  will  support  characterizing  the  state  of  software  engineering  practice  in  an  organization 
[1].  Like  any  tool,  however,  it  must  be  applied  in  the  proper  context  and  used  skillfully.  The 
primary  objectives  of  this  report  are  to  provide  guidance  on  creating  the  proper  environment  for  a 
meaningful  assessment  and  prescribing  effective  use  of  the  assessment  instrument. 

There  are  several  contexts  in  which  software  process  assessments  can  be  conducted.  Among 
these  are: 

•  Externally-assisted  assessments  of  an  organization,  such  as  by  the  SEI,  in  which  an 
assessment  team  conducts  in-depth  interviews  with  project  teams  and  formulates  a 
composite  profile  of  the  state  of  practice  in  the  organization.  The  SEI  has  performed 
assessments  of  this  type  and  is  continuing  to  do  so. 

•  Self-assessments  conducted  by  a  project  or  organization  in  order  to  determine  its 
state  of  software  engineering  practice.  There  are  a  number  of  disadvantages  to 
conducting  self-assessments  [2]. 

•  Workshop  assessments  conducted  in  conjunction  with  a  conference  or  other 
tutorials.  This  type  of  assessment  can  be  useful  in  quickly  gathering  industry  profile 
data,  generating  a  high  volume  of  feedback  on  the  quality  of  the  assessment  instru¬ 
ment,  and  providing  broader  awareness  of  the  assessment  process  and  its  benefits. 

The  SEI  will  be  conducting  assessments  of  this  type. 

•  Contractor  evaluation  assessments  conducted  as  part  of  the  DoD  procurement  proc¬ 
ess.  It  is  expected  that  the  assessment  instrument  being  developed  by  the  SEI  will 
be  used  in  various  phases  of  the  DoD  software  acquisition  process  but  the  SEI  will 
not  participate  in  such  assessments. 
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This  report  will  specifically  address  SEI-assisted  assessments  of  an  organization.  The  SEI  has 
used  its  assessment  instrument  to  conduct  trial  assessments  of  some  35  projects  in  ten  industrial 
and  government  software  organizations.  Early  results  indicate  that  an  SEI-developed  software 
process  maturity  model  [3]  reasonably  characterizes  the  state  of  software  engineering  practice 
within  a  software  organization  and  provides  a  mechanism  to  rapidly  identify  the  key  improvement 
issues  that  they  face. 

This  document  is  intended  to  serve  as  a  guide  for  conducting  SEI-assisted  software  process 
assessments.  Although  each  of  the  assessment  types  mentioned  above  may  require  slightly 
different  methods,  there  is  much  that  they  have  in  common.  Thus,  this  document  should  be 
helpful  to  those  who  are  charged  with  conducting  any  of  the  software  process  assessment  types 
mentioned  above.  As  the  SEI  gains  experience  in  conducting  assessments,  this  document  will  be 
updated  to  reflect  its  more  complete  understanding  of  how  to  effectively,  efficiently,  and  accu¬ 
rately  characterize  the  software  engineering  capability  of  organizations. 

1.1.  Overview  of  the  Assessment  Process 

Software  assessments  are  similar  to  the  organizational  development  process,  which  has  been 
used  successfully  to  motivate  change  in  a  number  of  very  large  organizations  [4,  5,  6].  In  both 
organizational  development  and  software  assessment,  the  crucial  prerequisite  is  to  gain  sufficient 
rapport  with  the  organization’s  key  people  so  they  will  share  their  problems,  concerns,  and  crea¬ 
tive  ideas. 

SEI-assisted  assessments  are  typically  conducted  in  five  phases: 

1 .  During  the  first  phase,  an  organization  is  identified  as  a  desirable  candidate  for 
assessment.  The  SEI  initiates  contact  and  obtains  organizational  commitment  to 
the  full  assessment  process.  This  commitment  includes  the  personal  participation 
of  the  senior  site  manager,  site  representation  on  the  assessment  team,  and  agree¬ 
ment  to  follow  up  on  recommended  actions.  An  assessment  agreement  (see  Ap¬ 
pendix  ),  which  includes  these  and  other  elements  of  the  joint  agreement,  is  signed 
by  the  organization  and  the  SEI. 

2.  The  second  phase  is  devoted  to  preparing  for  the  on-site  assessment.  An  assess¬ 
ment  team  is  selected  and  trained,  and  an  assessment  plan  is  formulated. 

3.  In  the  third  phase,  the  on-site  assessment  is  conducted.  On-site  presentations  are 
made  to  orient  site  personnel  who  will  be  participating  in  the  assessment.  The 
assessment  instrument  is  applied,  and  the  resulting  data  and  information  is  re¬ 
viewed  and  evaluated  by  the  assessment  team.  The  final  on-site  activity  is  to 
present  preliminary  findings  to  assessment  participants  and  senior  site  manage¬ 
ment. 

4.  The  fourth  phase  is  concerned  with  formulation  and  communication  of  final  assess¬ 
ment  findings  and  action  recommendations.  The  assessment  team  prepares  a  for¬ 
mal  written  report  which,  along  with  a  formal  on-site  briefing,  is  presented  to  the 
organization. 

5.  The  final  phase  consists  of  post-assessment  follow-up  activities.  An  action  team 
composed  entirely  of  professionals  from  the  organization  is  assembled  and  charged 
with  planning  and  implementing  the  recommendations.  Typically,  there  is  also 
some  continuing  support  and  guidance  by  the  SEI,  as  well  as  a  subsequent  follow¬ 
up  assessment  to  determine  the  overall  results. 
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1.2.  Assessment  Principles 

There  are  several  critical  prerequisites  to  a  successful  assessment  which  must  be  thoroughly 
understood  and  accepted  by  assessment  participants  In  both  ;he  assessed  and  assessing  organi¬ 
zations.  This  section  discusses  these  principles. 

1 .2.1 .  Confidentiality 

The  first  thing  to  establish  in  a  software  assessment  is  that  il  is  being  done  for  the  benefit  of  the 
organization.  This  Is  both  essential  and  extraordinarily  difficult  to  do.  The  assessment  results 
must  be  kept  in  strict  confidence.  No  leaks  can  occur,  even  to  the  organization's  chief  executive. 
While  everyone  will  agree  to  confidentiality  in  principle,  It  is  hard  to  maintain.  Senior  manage¬ 
ment  has  a  proper  interest  in  the  results;  but  if  the  members  of  the  organization  learn  that  they 
cannot  really  speak  in  confidence,  trust  will  be  lost  that  will  be  nearly  impossible  to  rebuild. 

Confidentiality  is  required  at  all  levels  of  the  organization.  Al  the  professionals  who  talk  must  be 
told  that  what  they  say  will  not  be  attributed  to  them.  Several  projects  should  always  be  reviewed 
at  the  same  time  and  each  of  the  project  managers  must  be  told  the  results  for  his  or  her  project 
but  no  others.  Site  management,  however,  must  only  be  given  a  composite  story  on  the  facility. 
This  approach  insures  that  no  project  or  individual  is  singled  out  as  having  unique  problems. 

1.2.2.  Senior  Management  Involvement 

The  senior  site  manager  is  the  person  who  sets  the  opera  :ional  priorities  for  the  organization. 
This  individual  must  be  personally  involved  in  the  assessment  process  and  the  follow-up  actions. 
This  specifically  includes  attendance  at  initial  and  final  on-sito  briefings.  He  or  she  sets  the  basic 
priorities  and  must  be  convinced  of  the  action's  importance  if  anything  is  to  happen.  This  involves 
sufficient  priority  to  assign  qualified  people  for  the  planning  arid  implementation. 

Without  participation  of  the  senior  site  manager,  the  review  cannot  be  successful.  The  people  at 
lower  levels  can  generally  handle  the  problems  during  normal  or  routine  operations,  but  lasting 
improvement  of  the  software  process  must  necessarily  concern  the  organization's  behavior  in  a 
crisis.  That  is  when  the  process  is  under  the  most  stress,  when  management  is  most  likely  to 
defer  all  nonessential  activities,  and  when  serious  disasters  are  most  likely  to  occur.  Since  crises 
are  common  in  the  software  business,  a  process  must  be  able  to  survive  a  crisis  to  be  useful. 

1.2.3.  Non-Adversarial  Attitude 

The  third  principle  is  that  the  process  be  non-threatening  and  focused  on  learning  and  under¬ 
standing.  An  assessment,  by  its  very  nature,  is  an  arrogant  activity.  A  group  of  remote  "experts" 
plans  to  review  a  large,  complex  organization  and  in  only  a  few  days  tell  them  what  they  are 
doing  wrong  and  what  they  should  do  to  improve.  Generally,  the  local  people  work  hard,  are 
dedicated  to  doing  good  work,  and  have  been  trying  to  improve  things  for  years.  They  may  be 
understandably  skeptical  of  such  studies  and  secretly  expect  the  whole  thing  to  fail. 

This  attitude  is  not  only  understandable,  but  quite  proper.  No  small  team  of  outside  experts  can 
hope  to  identify  the  most  critical  problems  in  any  reasonably  large  software  organization  in  a  short 
time.  Complex  problems  rarely  have  simple  answers,  and  the  subtleties  of  most  organizations 
are  far  too  intricate  for  any  group  to  fathom  in  a  few  days. 
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The  tacit  assumptions  must  be  that  the  people  are  not  dumb  or  misguided  and  that  they  have 
many  good  ideas.  If  the  assessment  team  arrives  thinking  they  have  all  the  answers,  the  local 
professionals  will  quickly  sense  it.  Their  natural  reaction  will  be  to  attempt  to  prove  that  these 
"experts”  are  not  so  smart  after  all.  This  leads  to  a  subtle  and  unspoken  hope  that  the  assess¬ 
ment  will  fail.  Under  these  conditions  it  often  does. 

The  power  of  the  assessment  process,  however,  is  that  it  taps  the  knowledge  and  creative  skills 
of  the  best  local  people.  If  they  can  be  convinced  to  contribute,  the  assessment  can  be  a  catalyst 
to  motivate  the  entire  organization  to  self  improvement.  But  it  will  only  work  when  it  is  seen  as  a 
help,  rather  than  as  a  threat. 

Any  assessment  team  that  comes  across  as  smug  or  having  all  the  answers  will  cause  immediate 
resentment.  A  highly  critical  attitude  or  a  lack  of  interest  in  local  views  and  opinions  can  be 
deadly.  When  good  work  is  found  it  should  be  recognized,  and  mistakes  and  oversights  should 
be  objectively  noted.  The  assessment  team  must  expect  that  any  suggestions  they  might  make 
will  have  to  be  tested  and  evaluated  by  the  local  people  to  see  if  they  will  really  work  in  practice. 
As  hard  as  it  is  to  achieve,  the  proper  attitude  is  one  of  humble  but  assured  professionalism. 

Even  when  the  assessment  team  behaves  in  a  thoroughly  circumspect  and  humble  manner, 
some  of  the  local  managers  and  professionals  will  resent  the  assessment  activity  and  not  partic¬ 
ipate  fully,  no  matter  how  carefully  the  assessment  has  been  planned  and  implemented.  How¬ 
ever,  if  the  team  really  believes  that  they  cannot  be  successful  without  the  wholehearted  help  and 
support  of  the  on-site  professionals,  this  will  be  recognized  and  most  of  the  people  will  respond. 

1.2.4.  Action  Orientation 

Finally,  the  entire  motivation  of  the  assessment  must  be  directed  toward  improvement.  The 
orientation  is  action,  so  the  questions  must  focus  on  defining  those  problems  that  need  to  be 
solved  right  away.  If  this  is  not  the  case,  the  assessment  may  make  the  situation  worse.  Prior  to 
the  assessment,  the  local  professionals  are  generally  aware  of  the  worst  problems  but  assumed 
management  is  not.  While  this  leads  them  to  view  management  as  somewhat  inept,  they  could 
always  assume  that  management  really  didn’t  understand  the  issues  and  so  couldn’t  be  expected 
to  address  them.  After  the  assessment,  this  is  clearly  no  longer  the  case.  A  structured  study  by 
a  team  of  experts  has  listened  to  some  of  the  most  competent  professionals  in  the  organization 
describe  the  state  of  practice  and  what  should  be  done  about  it.  This  is  then  personally  reported 
to  the  senior  management  in  a  written  report,  together  with  a  set  of  recommendations.  Any 
management  that  does  not  then  take  action  must  be  seen  as  either  incompetent  or  distracted  by 
other,  more  important  issues.  In  either  case,  the  morale  of  the  software  professionals  is  bound  to 
suffer.  Hence,  either  be  prepared  to  take  action  or  don’t  assess. 


2.  Selecting  Candidate  Organizations 

Listed  below  are  criteria  to  be  considered  during  the  process  of  selecting  an  organization  for 
software  assessment.  While  it  is  impossible  to  formulate  rules  that  apply  in  every  case,  these 
guidelines  can  be  helpful  in  making  reasonable  selection  decisions. 
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2.1.  Generic  Criteria 

Criteria  for  selecting  an  organization  for  assessment  depend  on  the  objectives  of  the  assessment 
initiator.  In  the  case  of  the  SEI,  the  goal  is  to  demonstrate  th  3  feasibility  and  value  of  the  assess¬ 
ment  approach  to  improving  the  software  process  in  the  DoD  community.  Thus,  an  organization 
should  have  the  following  attributes: 

1.  The  organization  has  software  of  sufficient  significancs  to  warrant  special  attention. 

2.  The  senior  site  manager  agrees  to  personally  particioate  by  being  present  for  the 
opening  on-site  assessment  briefing  and  the  on-siUi  review  of  final  findings  and 
recommendations;  the  manager  also  agrees  to  devel  Dp  appropriate  action  plans  in 
response  to  the  assessment  recommendations. 

3.  The  organization's  management  is  willing  to  sign  an  assessment  agreement  (see 
Appendix ). 

4.  Data  from  the  assessment  will  be  of  value  to  the  SEI. 

2.2.  DoD  Organization  Criteria 

In  keeping  with  its  mission,  the  SEI  balances  its  efforts  among  the  Services  and  considers  DoD 
priorities  and  needs  in  selecting  candidates  for  assessment. 

2.3.  Special  Considerations  for  Industrial  Assessments 

The  following  criteria  are  intended  to  guide  the  SEI  selectic  n  of  industry  participants  in  the  as¬ 
sessment  activity. 

1.  The  SEI  will  avoid  participating  in  assessments  of  projects  that  are  engaged  in  a 
competitive  phase  of  a  procurement. 

2.  If  all  criteria  are  equally  satisfied  by  more  than  one  contractor  (that  is.  If  we  have 
multiple  acceptable  invitations)  the  order  of  the  assessments  to  be  undertaken  will 
be  determined  considering  the  order  in  which  the  nvitations  were  received,  the 
relative  value  of  the  assessment  to  the  National  interest  and  to  the  SEI,  and  the 
level  of  prior  joint  activity  between  the  SEI  and  the  org  anization. 


3.  Preparing  For  The  Assessment 

A  successful  assessment  requires  careful  and  detailed  preparation  and  training.  This  section 
discusses  pre-site  visit  activities  that  are  intended  to  set  the  stage  for  a  smooth  and  orderly 
on-site  assessment. 

3.1.  Assessment  Team  Composition 

Once  the  organization  has  agreed  to  an  assessment,  the  assessment  team  must  be  selected  and 
trained.  The  members  should  be  experienced  software  developers  and,  wherever  possible,  at 
least  one  member  should  have  had  experience  In  each  phase  of  the  software  development  and 
maintenance  process.  Three  to  five  professionals  are  typicaly  adequate,  although  more  may  be 
used  if  desired.  A  team  with  more  than  ten  members  can  De  expensive,  intimidating  to  partic¬ 
ipants.  or  hard  to  manage.  The  SEI  learn  leader  must  have  had  prior  assessment  experience 
and  be  familiar  with  the  software  development  and  maintenance  process.  All  the  team  members, 
however,  should  be  capable  of  making  presentations  and  gathering  information  in  a  non¬ 
threatening  manner. 
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At  least  one  member  from  the  organization  being  assessed  should  participate  as  a  full  member  of 
the  team.  This  facilitates  the  planning  process,  provides  a  means  for  the  team  to  learn  about  the 
organization,  and  establishes  a  focal  point  for  later  action.  Since  the  local  member  is  so  crucial  to 
the  success  of  the  entire  effort,  the  senior  site  manager  should  personally  make  the  selection. 

3.2.  Assessment  Team  Training 

The  entire  assessment  team  should  participate  in  a  brief  training  program.  This  is  not  only  to 
familiarize  them  with  the  assessment  activity  but  also  to  build  a  coherent  team.  The  site 
member(s)  must  participate  in  the  training  and  help  to  develop  the  plan  for  the  on-site  assess¬ 
ment.  A  site  member  must  also  be  designated  as  the  coordinator  for  the  on-site  arrangements. 
Typically,  two  or  three  days  will  be  required  for  training,  and  the  team  leader  will  conduct  the 
training. 

In  addition,  the  team  members  must  agree  to  dedicated  participation  during  the  training  period, 
the  on-site  review,  and  the  wrap  up  meetings.  This  means  that  phone  calls  will  be  held,  all 
meetings  and  other  commitments  rescheduled,  and  the  members  will  be  on  time  to  every  ses¬ 
sion. 

The  assessment  team  training  consists  of  the  following  sessions  (see  appendix  for  an  example 
training  agenda): 

1.  SEI  Overview  —  The  first  training  session  is  devoted  to  an  overview  of  the  SEI. 

This  is  done  to  familiarize  the  site  representative(s)  with  the  role  of  the  SEI  within 
the  field  of  software  engineering.  The  major  program  areas  are  briefly  identified 
and  reviewed. 

2.  Process  Management  Overview  —  The  foundational  material  dealing  with  process 
management  is  introduced  in  this  session  [3].  The  process  management  premise 
and  the  principles  of  process  management  are  discussed.  The  process  manage¬ 
ment  maturity  level  model  [3]  is  presented  and  used,  along  with  the  idea  of  tech¬ 
nology  stages,  to  provide  the  basis  for  the  process  technology  matrix  [1],  The 
significance  of  the  various  regions  in  the  process  technology  matrix  is  explained. 
Finally,  these  concepts  and  models  are  used  to  characterize  the  improvement  proc¬ 
ess. 

3.  Site  Overview  —  It  is  important  for  the  SEI  members  of  the  assessment  team  to 
understand  the  mission  and  function  of  the  site  to  be  assessed.  This  session  is 
devoted  to  a  discussion  of  the  organization  which  will  be  assessed.  This  presen¬ 
tation  is  given  by  the  senior  site  assessment  team  member. 

4.  Assessment  Introduction  and  Guidelines  —  This  training  session  is  spent  in  an 
overview  of  the  assessment  process.  The  guidelines  for  assessment  are  dis¬ 
cussed,  use  of  the  assessment  instrument  is  explained,  the  role  of  supporting 
materials  is  discussed,  and  assessment  evaluation  is  explained. 

5.  Assessment  Questionnaire  Review  —  The  assessment  is  based  on  responses  to 
the  assessment  instrument.  Therefore,  it  is  important  for  all  assessment  team 
members  to  fully  understand  the  spirit  and  intent  of  all  questions.  Thus,  this  ses¬ 
sion  is  devoted  to  a  detailed  review  of  each  question.  A  simulated  assessment  is 
conducted  based  on  previous  experiences  of  team  members. 

6.  Supporting  Materials  Discussion  —  It  is  important  to  be  able  to  verify  the  accuracy 
of  the  responses  to  the  SEI  questionnaire.  This  is  achieved  through  the  requests 
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for  supporting  materials  which  serve  to  validate  responses  to  the  questionnaire. 
During  this  session,  the  process  for  identifying  areas  for  further  investigation  is  re¬ 
viewed  and  particular  questions  are  identified  based  on  assessment  responses. 

7.  Assessment  Evaluation  and  Findings  —  This  session  is  devoted  to  a  discussion  of 
the  techniques  to  be  used  in  determining  both  project  and  composite  maturity  levels 
and  technology  ievels  [1,  3].  The  data  from  the  simulated  assessment  is  used  as 
the  basis  for  evaluation  and  preliminary  findings. 

8.  Conclusions  and  Recommendations  —  The  general  approach  to  formulating  con¬ 
clusions  and  recommendations  from  the  assessment  evaluation  and  findings  is  dis¬ 
cussed  in  this  session. 

9.  Planning  for  Site  Visit  —  Details  of  the  plan  for  the  site  visit  are  discussed.  The 
plan  for  the  on-site  assessment  itself  should  include  the  identification  of  the  specific 
projects  to  be  examined,  the  people  to  be  interviewed,  and  the  facilities  required. 
Since  the  people  and  facilities  in  most  software  organizations  are  heavily  com¬ 
mitted,  significant  advance  notice  is  generally  requirec. 

Typically,  three  to  five  projects  are  selected  as  representative  of  the  process  used 
at  the  location.  Multiple  projects  are  studied  both  to  assure  reasonable  coverage  of 
the  work  at  the  location  and  to  permit  presentation  of  composite  findings  In  the  final 
report  without  violating  the  confidentiality  principle. 

Improper  seating  can  send  subtle  and  erroneous  met  sages  to  the  Interviewees.  A 
conscious  effort  should  be  made  to  avoid  configurations  of  tables  and  chairs  which 
seem  to  put  the  assessment  team  in  a  superior  role.  Try  to  arrange  for  an  informal 
seating  arrangement  and  mix  the  assessment  team  members  with  the  site  person¬ 
nel. 

Daily  agendas  of  the  visit  should  be  discussed  and  agreed  upon  (See  Appendix  for 
an  example  on-site  agenda). 


4.  Conducting  the  Assessment 

This  section  discusses  the  sequence  of  activities  that  constitute  the  on-site  portion  of  the  assess¬ 
ment.  Typically,  these  activities  require  about  four  days  to  complete. 

4.1.  Introductory  Presentation 

The  assessment  starts  with  a  presentation  to  the  senior  site  manager,  his  or  her  immediate  staff, 
the  managers  of  the  selected  projects,  as  well  as  senior  soitware  professionals  from  these  and 
other  projects  who  are  most  knowledgeable  on  particular  facets  of  the  development  and  mainte¬ 
nance  process.  The  presentation  opens  with  a  brief  stateme  it  of  the  agenda  and  an  introduction 
of  the  assessment  team  members.  Next,  the  assessment  agreement  is  reviewed,  including  a 
description  of  how  the  assessment  will  be  conducted  and  the  schedule  for  the  assessment  and 
following  activities.  Any  questions  or  concerns  are  addressed,  and  copies  of  the  assessment 
agreement  and  schedules  are  distributed. 
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4.2.  Applying  the  Assessment  Instrument 

Following  the  introductory  presentation,  the  assessment  team,  project  managers,  and  technical 
participants  meet  in  a  joint  session  devoted  to  a  more  detailed  discussion  of  the  assessment 
process.  This  is  followed  by  an  interactive  period  devoted  to  formulating  initial  responses  to  the 
questions  in  the  assessment  instrument.  Project  managers  mark  their  copies  of  the  assessment 
instrument,  and  the  assessment  team  provides  clarification  on  questions  as  necessary. 

Following  this  joint  session,  the  assessment  team  meets  in  a  closed  session  during  which  an 
initial  determination  of  the  maturity  level  of  the  projects  and  organization  is  made.  Also,  areas 
where  supporting  materials  are  appropriate  are  identified. 

Next,  the  assessment  team  meets  with  the  project  representatives  on  a  project-by-project  basis  to 
ask  additional  questions  in  the  areas  previously  identified  and,  where  appropriate,  to  request 
specific  supporting  materials. 

As  a  final  activity  for  this  stage,  the  assessment  team  holds  a  private  wrap-up  meeting  to  review 
the  progress  of  the  assessment  and  to  prepare  for  the  next  stage. 

4.3.  Functional  Areas  Interviews 

Next,  interviews  are  conducted  with  project  personnel  from  each  major  functional  area  of  the 
organization’s  software  process.  Approximately  six  separate  interview  sessions  are  held  with  the 
format  varying  between  45  minute  sessions  with  an  individual  and  90  minute  sessions  with  small 
groups.  The  entire  assessment  team  is  present  for  these  interviews.  The  purpose  of  the  interview 
sessions  is  to  determine  the  actual  details  of  the  software  process  since  the  assessment  ques¬ 
tionnaire  only  provides  an  overview  of  the  process.  Each  individual  to  be  interviewed  is  told  they 
may  bring  1  or  2  foils  if  they  wish,  but  that  it  is  to  be  an  informal  session  during  which  they  will  be 
asked  to  describe  how  they  do  their  work.  Where  possible,  they  should  bring  some  typical  work 
products  to  demonstrate  the  methods  they  use.  Near  the  end  of  the  interview,  a  key  question 
should  be  asked:  "If  there  was  one  thing  you  could  do  to  improve  the  quality  or  productivity  of 
your  work,  what  would  it  be?"  The  answers  to  this  question  are  very  important  because  experi¬ 
ence  shows  that  the  people  working  on  the  project  generally  know  better  than  anyone  what  needs 
to  be  done  to  improve  it. 

As  the  final  item,  the  assessment  team  holds  a  closed  wrap-up  meeting  to  review  the  day’s 
findings  and  prepare  for  the  next  stage. 

4.4.  Project  Feedback  Reviews 

The  assessment  team  meets  with  project  representatives  separately  to  review  the  supporting 
materials  of  each  project.  At  this  time,  the  assessment  team  will  verbally  review  its  findings  for 
the  project  and  the  composite  results  for  the  organization  and  allow  the  project  representatives  to 
comment  on  them. 

Ad  hoc  discussions  with  various  groups  may  occur  following  the  individual  project  reviews.  The 
intent  here  is  to  gather  additional  data  to  support  the  findings  and  to  identify  any  other  problem 
areas. 
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At  the  conclusion  of  this  stage,  the  assessment  team  shou  d  have  achieved  consensus  on  the 
final  findings  for  presentation  at  a  joint  meeting  on  the  last  da^  of  the  assessment. 

4.5.  Presentation  of  Findings 

This  stage  is  devoted  to  preparation  and  presentation  of  individual  project  and  composite  organi¬ 
zation  findings.  The  assessment  team  meets  privately  to  review  their  conclusions,  make  changes 
they  deem  warranted  based  on  review  of  the  supporting  materials,  decide  on  the  key  recommen¬ 
dations  to  be  made,  and  prepare  the  report  for  presentation  at  the  final  management  meeting. 
Some  key  items  to  consider  for  the  final  presentation  are: 

1.  The  cooperation  and  support  of  the  participants  should  be  acknowledged,  and  any 
unusually  good  or  innovative  work  should  be  icentified  (by  name  wherever 
appropriate). 

2.  Only  the  composite  conclusions  of  the  assessment  <re  presented,  so  no  project  is 
uniquely  identified  with  any  problem. 


5.  Final  Assessment  Findings  and  Recommendations 

Following  the  on-site  assessment  visit,  the  assessment  team  prepares  a  draft  final  report  of  the 
assessment  findings  and  recommendations.  The  recommer  dations  should  be  limited  to  at  most 
ten  items,  with  the  four  or  five  most  urgent  highlighted  for  immediate  attention.  No  organization 
can  handle  more  than  a  few  high  priority  tasks  at  one  time,  so  the  items  requiring  the  most  urgent 
attention  should  be  clearly  spelled  out.  The  organization’s  assessment  team  members  then  visit 
the  SEI  for  the  purpose  of  reviewing  the  draft  report.  Once  consensus  within  the  assessment 
team  has  been  achieved,  the  next  on-site  action  is  the  presentation  of  the  final  report  to  the 
senior  site  manager  and  staff.  This  typically  occurs  about  a  month  after  the  completion  of  the 
on-site  portion  of  the  assessment. 

The  final  report  should  contain  the  following  information: 

1 .  Summary  and  Conclusion  —  an  executive  level  summary  of  the  assessment  team’s 
findings. 

2.  Software  Assessment  —  a  brief  description  of  the  context  of  the  assessment  and  a 
chronology  of  key  events. 

3.  Site  Status  —  in  terms  of  composite  maturity  level  and  technology  state. 

4.  Key  Findings  —  the  most  important  findings. 

5.  Recommendations  —  the  assessment  team’s  recommendations. 

6.  Assessment  Agreement  —  a  copy  of  the  signed  as  ;essment  agreement  included 
as  an  appendix. 
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6.  Post-Assessment  Follow-Up 

Although,  strictly  speaking,  the  goal  of  an  assessment  is  to  accurately  characterize  the  current 
state  of  software  engineering  practice  in  the  organization  and  identify  key  next  steps  for  improve¬ 
ment,  the  ultimate  intent  is  to  be  the  catalyst  for  improvement  in  the  assessed  organization.  This 
section  focuses  on  activities  that  take  place  after  the  assessment  and  which  can  be  effective  in 
increasing  the  likelihood  that  improvements  will  occur. 

6.1.  Action  Plans 

Following  the  on-site  presentation  of  final  findings  and  action  recommendations,  action  plans  are 
prepared  by  the  assessed  organization,  generally  under  the  guidance  of  the  site  assessment 
team  member  who  was  named  for  this  purpose  by  the  senior  site  manager.  If  properly  chosen, 
he  or  she  is  now  fully  knowledgeable  of  the  issues  to  be  addressed  and  known  and  respected  by 
the  people  who  must  be  involved.  It  is  generally  wise  to  involve  leading  professionals  from  the 
various  projects  in  this  work.  This  not  only  helps  to  produce  a  quality  result  but  it  also  facilitates  its 
ultimate  acceptance. 

The  action  plans  should  focus  on  the  sub-goals  identified  for  periodic  management  review  (see 
Section  6.2).  Only  one  or  two  action  plans  should  be  attempted  at  a  time,  and  special  attention 
should  be  devoted  to  ensuring  early  small  successes  that  will  pave  the  way  for  larger  successes 
as  the  effort  builds  momentum  via  acceptance  and  participation  within  the  organization. 

A  timetable  for  staffing  the  action  plan  team  and  developing  the  action  plan  should  be  completed 
within  2  to  3  weeks  of  the  on-site  presentation  of  the  final  assessment  report.  Periodic  and 
frequent  (e.g.,  weekly)  senior  management  updates  should  occur  until  the  action  plan  team  is 
fully  staffed  and  functioning.  This  safeguard  ensures  that  the  proper  priority  is  given  to  the 
important  job  of  formulating  concrete  improvement  plans. 

6.2.  Periodic  Site-Management  Review 

One  of  the  risks  with  assessments  is  that,  after  they  are  completed,  a  few  superficial  or  ineffective 
efforts  will  be  made  to  initiate  improvement  and  then  everything  will  revert  back  to  business  as 
usual.  Some  catalyst  is  required  to  maintain  focus  on  the  improvement  process.  One  highly 
effective  means  is  periodic  management  review.  This  can  be  done  by  establishing  a  clear  set  of 
goals  to  be  achieved  by  the  time  of  the  second  assessment  and  then  defining  sub-goals  to  be 
reached  in  each  of  the  intervening  quarters.  Quarterly  reviews  are  then  scheduled  with  senior 
management  with  the  objectives  defined  in  advance.  This  will  not  only  help  to  crystalize  the  plan 
but  will  also  serve  to  create  those  periodic  crises  which  are  required  to  maintain  priority  and 
focus. 

6.3.  Process  Group  Formation 

It  is  expected  that  some  resources  will  be  required  to  manage  the  continuing  improvement  of  the 
organization’s  software  process.  Software  engineering  process  groups  provide  a  center  of  exper¬ 
tise  to  assist  the  software  practitioners,  to  act  as  a  focal  point  for  improvement,  and  to  serve  as 
the  vehicle  for  organizational  learning.  A  small  but  dedicated  group  of  very  competent  and  expe¬ 
rienced  professionals  is  needed.  Typically  1 .5%  to  4%  of  the  organization  is  adequate. 
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A  software  process  group  typically  concerns  itself  with  the  fol  owing  aspects  of  the  software  proc¬ 
ess: 

•  defining  and  documenting  the  software  development  process, 

•  establishing  critical  process  measures, 

•  planning  for  new  tools  and  technologies, 

•  conducting  quarterly  process  reviews  for  senior  managsment. 

•  assisting  the  projects  in  applying  improved  methods  and  practices. 

Additional  areas  of  activity  may  be  appropriate,  but  they  will  depend  on  the  specific  findings  and 
recommendations  of  the  assessment  team. 

6.4.  SEI  Continuing  Support  &  Guidance 

Typically,  the  SEI  will  be  willing  and  able  to  provide  post-assossment  support  and  guidance  to  an 
assessed  organization.  The  extent  and  nature  of  this  invDlvement  will  be  determined  on  an 
individual  basis. 

6.5.  Reassessments 

Organizations  should  generally  plan  to  conduct  a  follow-up  Reassessment  about  a  year  or  so 
after  the  action  plans  have  been  developed  and  approved.  T iis  is  important  for  several  reasons: 

•  to  assess  the  progress  that  has  been  made, 

•  to  establish  a  target  for  completion  of  the  most  important  actions, 

•  to  establish  new  priorities  for  continued  improvement. 
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Software  Engineering  Assessment  Agreement 


This  Agreement  is  entered  into  by  and  between  the  Software  Engineering  Institute  (SEI)  and 

_ (hereinafter  refered  to  as  Affiliate)  with  its  principal 

office  located  at _ .  This  Agreement  will  be  effective  when  executed  by 

both  parties.  Its  duration  will  be  limited  by  the  nature  and  scope  of  the  undertakings  set  forth  in 
the  Agreement.  Certain  commitments  will  survive  completion  of  the  basic  tasks  set  forth. 

The  SEI  is  a  Federally  Funded  Research  and  Development  C  enter  (FFRDC)  owned  and  operated 
by  Carnegie  Mellon  University  pursuant  to  contract  with  the  Department  of  Defense  (DoD).  In¬ 
cluded  in  the  SEI  Charter  is  a  requirement  that  the  SEI  shall  sstablish  standards  of  excellence  for 
software  engineering  practice.  In  furtherance  of  that  requirement,  the  SEI  is  conducting  assess¬ 
ments  of  software  engineering  practice  with  individual  companies  and  studying  industry-wide 
practices  of  software  engineering  (the  Assessment  Program).  Affiliate  has  a  vested  interest  in 
assessing  its  level  of  software  engineering  practice  on  both  an  absolute  and  comparative  basis. 

Therefore  the  parties  hereto  agree  as  follows: 

1.  Affiliate  agrees  to  participate  with  the  SEI  in  an  assessment  of  the  Affiliate’s  soft¬ 
ware  engineering  practices. 

2.  The  specific  results  of  the  Affiliate  assessment  shall  l>e  proprietary  to  Affiliate  to  be 
used  by  Affiliate  as  it  chooses  (with  appropriate  credit  to  the  SEI)  subject  to  the  right 
of  the  SEI  to  use  such  results  as  hereinafter  described. 

3.  Confidentiality  is  essential  to  the  success  of  the  SE I  Assessment  Program.  The 
SEI  will  not  identify  organizations  participating  in  the  Assessment  Program. 

4.  The  SEI  is  free  to  use  assessment  data  and  conclusions  to  be  derived  therefrom  for 
statistical,  analytical  or  reporting  purposes  provided  lhat  the  confidentiality  require¬ 
ment  can  be  honored  and  that  the  information  can  bo  used  without  attribution  to  its 
source  either  directly  or  by  inference. 

5.  The  SEI  will  not  publish  collective  data  externally  umess  such  data  is  based  upon 
assessments  from  not  less  than  1 0  different  organizations. 

6.  The  senior  manager  of  the  segment  of  Affiliate  to  b€  assessed  will  actively  partic¬ 
ipate  in  the  assessment  by  agreeing  to  be  present  (on  mutually  acceptable  dates) 
for  the  opening  on-site  assessment  briefing  and  the  on-site  review  of  final  findings 
and  recommendations.  This  level  of  participation  is  critical  to  the  success  of  the 
project. 

7.  The  SEI  will  provide  four  to  six  professionals  as  assessment  team  members. 

8.  The  Affiliate  will  provide  one  or  two  professionals  as  sssessment  team  members.  It 
is  expected  that  one  of  these  professionals  will,  in  addition,  be  assigned  the  follow¬ 
ing  responsibilities: 

a.  participate  in  a  one  or  two  day  working  session  at  the  SEI  shortly  following 
the  on-site  assessment  period  to  prepare  the  final  assessment  report. 

b.  develop  the  organization’s  action  plan  based  jpon  the  final  assessment  re¬ 
port. 
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9.  There  will  be  two  days  of  pReassessment  training  at  the  SEI  for  the  entire  assess¬ 
ment  team. 

1 0.  The  assessment  by  the  SEI/Affiliate  team  at  the  Affiliate  site  will  be  done  in  three  or 
four  days. 

1 1 .  Typically,  three  to  five  projects  will  be  reviewed  during  the  on-site  assessment 
period.  The  Affiliate  will  ensure  that  none  of  the  projects  selected  for  review  is 
involved  in  a  competitive  phase  of  a  procurement  during  the  on-site  assessment 
period. 

12.  The  SEI  as  soon  as  practicable  thereafter  will  provide  to  Affiliate  a  final  assessment 
report  which  will  include  recommended  actions.  The  SEI  does  not  represent  any 
U.S.  Government  procurement  agency;  therefore,  the  recommendations  made  by 
the  SEI  are  not  to  be  construed  as  contractual  directions  or  constructive  changes  to 
contracts  which  may  exist  between  the  affiliate  and  the  U.S.  Government. 

13.  The  Affiliate  will  within  60  days  thereafter  provide  a  review  for  the  SEI  of  its  action 
plan  based  on  the  final  assessment  report.  The  Affiliate  is  the  sole  determinant  as 
to  the  extent  that  recommended  actions  are  implemented  through  its  action  plan.  It 
is  expected  that  the  review  will  cover  the  reasons  for  rejection  of  any  final  assess¬ 
ment  report  recommended  actions  not  incorporated  into  the  action  plan. 

14.  The  SEI,  at  its  option,  and  with  the  consent  of  the  Affiliate,  may  continue  periodic 
interaction  with  the  Affiliate  during  implementation  of  the  action  plan. 

1 5.  Each  participating  organization  will  be  responsible  for  its  personnel  and  their  ex¬ 
penses.  Each  will  provide  to  the  other  reasonable  working  space  and  support  ser¬ 
vices  for  assessment  team  activities. 

1 6.  If  Affiliate  discloses  identified  proprietary  information  to  the  SEI,  such  information 
should  be  disclosed  pursuant  to  a  separate  nondisclosure  agreement.  The  execu¬ 
tion  of  such  agreement  notwithstanding,  the  parties  agree  to  respect  the  confiden¬ 
tial  nature  of  this  Agreement  and  all  exchanges  of  information  hereunder.  All  con¬ 
fidential  and  proprietary  information  shall  be  treated  with  at  least  the  degree  of  care 
with  which  each  party  treats  its  own  such  information. 

1 7.  When  Affiliate  has  received  an  assessment  report  from  the  SEI  and  has  in  turn 
reviewed  its  action  plan  with  the  SEI,  the  basic  assessment  tasks  will  be  complete 
and  this  Agreement  will  have  served  its  primary  purpose.  However,  any  commit¬ 
ments  which  by  their  nature  are  intended  to  survive  termination  of  the  Agreement 
(such  as  nondisclosure)  will  survive. 

18.  This  agreement  shall  not  constitute,  create,  give  effect  to,  or  otherwise  imply  a  joint 
venture,  partnership  or  formal  business  organization  of  any  kind.  Each  party  to  this 
Agreement  shall  act  as  an  independent  contractor  and  not  as  an  agent  for  the 
other,  and  neither  party  shall  have  any  authority  to  bind  the  other  except  to  the 
extent,  if  any,  specifically  provided  herein  or  by  other  written  mutual  agreement  of 
the  parties. 


Software  Engineering  Institute 

By _ 

Date _ 


Affiliate 

By _ 

Date 
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Appendix  B:  Example  Assessment  Training  Agenda 


Agenda  Overview 

•  Software  Engineering  Institute  Overview 

•  Process  Management  Overview 

•  Site  Overview 

•  Assessment  Process  Overview  and  Guidelines 

•  Assessment  Questionnaire  Review 

•  Supporting  Materials  Discussion 

•  Assessment  Evaluation  and  Findings 

•  Conclusions  and  Recommenc  ations 


Day  1  Agenda 


9:00-9:15  am 

Introductions 

9:15-9:30  am 

SEI  Overview 

9:30-10:15  am 

Process  Manage  nent  Overview 

10:15-10:30  am 

Break 

10:30-11:30  am 

Site  Overview 

11:30-12:15  pm 

Assessment  Process  Overview  and  Guidelines 

12:15-1 :00  pm 

Lunch 

1 :00-  3:00  pm 

Questionnaire  Review 

3:00-3:15  pm 

Break 

3:15-5:00  pm 

Supporting  Matei  ials  Discussion 

5:00  pm 

Adjourn 

Day  2  Agenda 


9:00-10:15  am 

Assessment  Eva  uation  and  Findings 

10:15-10:30  am 

Break 

10:30-11:30  am 

Conclusions  and  Recommendations 

11:30-12:30  pm 

Lunch 

12:30-2:30  pm 

Planning  for  On-Site  Visit 

2:30-3:00  pm 

Review  of  Training  Session  Operation 
and  Effectiveness 

3:00  pm 

Adjourn 
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Appendix  C:  Example  Detailed  On-Site  Agenda 

First  Day  Activities 

Purpose:  Kick  off  the  assessment,  administer  the  questionnaire.  By  the  end  of  the  day,  we 
should  be  able  to  request  supporting  materials  and  know  approximately  where  the  organization 
stands. 

8:00  am  -  9:15  am  Introductory  Management  Meeting 

Attendees:  Senior  Site  Manager 

Senior  Management  Team 

Project  Managers  and  Technical  Participan  s 

Assessment  Team 

Presentation  Materials: 

1 .  Introductions 

2.  SEI  Overview 

3.  Process  Management  Overview 

4.  Assessment  Agreement 

5.  Detailed  Agenda 

Facilities:  Large  conference  room,  overhead  projector,  whiteboard,  markers  and  erasers 
9:15  am  -  9:30  am  Break 

9:30  am  - 12:00  noon  Assessment  Overview  and  Questionnaire 

Attendees:  Project  Managers  and  Technical  Participarts 
Assessment  Team 

Presentation  Materials: 

1 .  Assessment  Process  Overview 

2.  Questionnaire 

Facilities:  Large  conference  room,  overhead  projectcr,  whiteboard,  markers  and  erasers, 
questionnaire  recording  forms,  extra  pens  or  pencils  with  erasers 

12:00  noon  -  1:00  pm  Lunch 

1:00  pm  -  2:30  pm  Closed  Session 
Attendees:  Assessment  Team 

Objectives: 

1.  Go  through  the  process  of  determining  the  maturity  level  of  the  organization 
by  examining  the  level  2  and  3  asterisked  questions  for  each  project  and  for 
the  organization  as  a  whole. 

2.  Determine  where  probing  is  appropriate  for  supporting  materials  from  the 
various  projects. 
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Facilities:  Private  meeting  room,  forms  for  recording  level  2  and  3  asterisked  questions  for 
each  project  and  summarily,  reproduction  facilities,  whiteboard  and  markers 

2:30  pm  -  2:45  pm  Break 

2:45  pm  -  4:45  pm  Supporting  Materials  Discussion 

Attendees:  Project  Managers  and  Technical  Participants 
Assessment  Team 

Objectives: 

1 .  Spend  20  minutes  per  project  poking  at  "issue"  areas 

2.  Where  appropriate,  request  specific  supporting  materials  to  be  provided  on 
day  3. 

Facilities:  Conference  room,  whiteboard  and  markers 
4:45  pm  -  5:00  pm  Break 

5:00  pm  -  6:30  pm  First  Day  Wrap-Up  Meeting 
Attendees:  Assessment  Team 

Objectives: 

1 .  Discuss  how  the  assessment  is  proceeding. 

2.  Identify  and  clarify  what  the  real  issues  are. 

Facilities:  Private  meeting  room,  whiteboard  and  markers 
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Second  Day  Activities 

Purpose:  The  second  day  activities  serve  as  a  reality  check.  While  the  participants  from  the  first 
day  are  given  an  opportunity  to  collect  the  requested  support  materials,  the  assessment  team 
meets  with  other  groups  to  learn  the  views  and  practices  at  some  of  the  leading  professionals  in 
the  organization  and  to  see  if  any  significant  discrepancies  exist  between  actual  practice  and  the 
questionnaire  responses.  The  assessment  team  will  hold  five  separate  informal  technical  discus¬ 
sions,  each  with  3-5  technical  professionals  with  expertise  In  the  particular  area  being  discussed. 
It  should  be  emphasized  that  no  managers  should  be  presen  :  during  any  of  these  meetings.  The 
professionals  selected  to  participate  should  have  the  following  characteristics: 

1 .  Knowledgeable  in  the  area  being  discussed 

2.  Respected  In  the  organization 

3.  Non-managerlal 

4.  Willing  to  speak  their  minds 

5.  Not  necessarily  from  one  of  the  projects  being  assessed 

Each  participant  should  be  selected  in  advance,  and  should  be  given  a  set  of  guidelines  in  ad¬ 
vance  which  contain  the  following  information: 

1 .  Summary  of  why  the  discussions  are  being  held 

2.  Rules  for  the  discussions 

3.  Emphasis  on  confidentiality 

7:30  am  -  9:15  am  Quality  Assurance  and  Release 

9:15  am  - 1 1:00  am  System  Test  and  Integration 

1 1:00  am  - 12:00  noon  Lunch 

12:00  noon  -  1:45  pm  Software  Development 

1:45  pm  -  3:30  pm  Software  Design 

3:30  pm  -  5:15  pm  Systems  Engineering 

5:15  pm  -  5:30  pm  Break 

5:30  pm  -  7:00  pm  Second  Day  Wrap-up  Meeting 
Attendees:  Assessment  Team 

Objectives: 

1.  Compare  what  was  learned  today  with  questionnaire  responses,  and  deter¬ 
mine  whether  or  not  they  support  one  another. 

2.  Establish  a  clear  focus  for  further  in-depth  exarrination. 

Facilities:  Private  meeting  room,  whiteboard  and  markers,  reproduction  facilities 
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Third  Day  Activities 

Purpose:  Hold  reviews  with  each  assessed  project  to  look  at  requested  support  materials,  ver¬ 
bally  state  our  findings,  and  allow  the  project  leaders  to  give  us  feedback  or  refute  our  findings. 

7:30  am  -  9:00  am  Feedback  Reviews 

Attendees:  Project  Managers  (individually) 

Assessment  Team 

Objectives: 

1 .  Spend  45  minutes  per  project  doing  the  things  outlined  in  the  purpose  for  the 
day. 

Facilities:  Small  conference  or  meeting  room,  whiteboard  and  markers 
9:00  am  -  9:15  am  Break 

9:15  am  - 10:45  am  Feedback  Reviews  (continued) 

10:45  am  - 1 1:00  am  Break 
1 1:00  am  -  12:30  pm  Feedback  Reviews  (continued) 

12:30  pm  - 12:45  pm  Make  Ad  Hoc  Arrangements 
Objectives: 

1 .  Place  phone  calls  to  inform  various  groups  of  our  intent  to  talk  with  them 
about  their  operation  later  in  the  afternoon. 

Facilities:  Telephones,  telephone  directory 
12:45  pm  -  1:45  pm  Lunch 

1:45  pm  -  3:15  pm  Ad  Hoc  Discussions 
Objectives: 

1 .  Conduct  meetings  as  necessary  to  prepare  for  formulation  of  final  findings. 

Facilities:  No  special  requirements 
3:15  pm  -  3:30  pm  Break 

3:30  pm  -  no  later  than  8:00  pm  Third  Day  Wrap-up  Meeting 
Attendees:  Assessment  Team 

Objectives: 

1.  Put  together  our  final  findings  which  will  be  presented  at  the  final  manage¬ 
ment  meeting  on  the  fourth  day. 

Facilities:  Private  meeting  room,  word  processor  or  workstation,  printing  facilities,  slide¬ 
making  and  reproduction  facilities 
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Fourth  Day  Activities 

Purpose:  Work  on  individual  project  feedback,  give  composite  feedback  to  project  managers, 
work  on  outline  of  final  report,  and  give  composite  feedback  1 5  senior  management. 

7:30  am  -  9:00  am  Closed  Session 
Attendees:  Assessment  Team 

Objectives: 

1 .  Prepare  individual  project  feedback  which  the  site  assessment  team  member 
can  then  discuss  with  the  individual  project  managers  after  the  conclusion  of 
the  assessment. 

Facilities:  Private  meeting  room,  reproduction  facilities 

9:00  am  - 10:30  am  Composite  Feedback 

Attendees:  Project  Managers 
Assessment  Team 


Objectives: 

1.  Present  the  composite  status  of  the  organization  to  the  project  managers. 
Should  be  able  to  use  the  same  set  of  slides  which  will  be  presented  to 
senior  management  later. 

i 

Facilities:  Small  conference  room 
10:30  am  - 12:30  pm  Closed  Session 
Attendees:  Assessment  Team 

Objectives: 

1 .  Prepare  an  outline  of  the  final  report  by  developing  a  set  of  action  recommen¬ 
dations.  This  material  will  not  be  presented  at  this  time,  but  will  be  expanded 
on  later  to  form  the  final  report  to  the  organizatbn. 

Facilities:  Private  meeting  room,  whiteboard  and  markers,  reproduction  facilities,  overhead 
projector 

i 

12:30  pm  - 1:00  pm  Lunch 

1:00  pm  -  2:30  pm  Final  Management  Meeting 

Attendees:  Senior  Site  Manager 

Senior  Management  Team 
Assessment  Team 


Objectives: 

1.  Present  the  findings  which  have  been  made  o/er  the  past  three  days.  Brief 
management  on  the  next  steps. 

Facilities:  Large  conference  room,  overhead  projector 
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studies  and  implementation  of  the  studies  are  described. 

TROY,  R.  and  C.  Baluteau,  Assessment  of  Software  Quality  for  the  AIRBUS  A-310  Automatic 
Pilot,  Paper  presented  at  the  15th  Annual  International  Symposium  on  Fault  Tolerant  Software, 
1985. 


30 


CMU/SEI-87-TR-16 


As  for  the  whole  of  the  Automatic  Flight  Control  System,  the  automatic  pilot  for  the  AIRBUS  is  a 
computer  system  with  high  criticality.  In  order  to  assess  the  quality  of  the  of  the  software,  a 
number  of  observations  and  measurements  have  been  performed  during  the  development  cycle. 
This  paper  presents  a  general  method  of  evaluation,  the  data  collected,  and  some  evaluation 
results. 
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Formal  technical  reviews  supply  the  quality  measurement  to  the  "cost  effectiveness"  equation  in  a 
project  management  system.  There  are  several  unique  formal  technical  review  procedures,  each 
applicable  to  particular  types  of  technical  material  and  to  tho  particular  mix  of  the  Review  Com¬ 
mittee.  All  formal  technical  reviews  produce  reports  on  the  overall  quality  for  project  manage¬ 
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toric  account  of  the  systems  development  process.  Historic  origins  and  future  trends  of  formal 
and  informal  technical  reviews  are  discussed. 
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